Teaching Bandits How to Behave Manuscript

Consider a setting in which an agent selects an action in each time period and there is an interested party who seeks to induce a particular action. The interested party can associate incentives with actions to perturb their value to the agent. The agent’s decision problem is modeled as a multi-armed bandit process where the intrinsic value for an action updates independently of the state of other actions and only when the action is selected. The agent selects the action in each period with the maximal perturbed value. In particular, this models the problem of a learning agent with the interested party as a teacher. For inducing the goal action as soon as possible, or as often as possible over a fixed time period, it is optimal for an interested party with a per-period incentive budget to assign the budget to the goal action and wait for the agent to learn to want to make that choice. Teaching is easy in this case. In contrast, with an across-period budget, no algorithm can provide good performance on all instances without knowledge of the agent’s update process, except in the particular case in which the goal is to induce the agent to select the goal action